This switches us away from the code signing certificate expiring in
June, to a new one valid until 2027.

https://hg.mozilla.org/mozilla-central/rev/70c61aaf0509

Backed out out of central as requested by bhearsum for causing Bug 1894138: https://hg.mozilla.org/mozilla-central/rev/b7a1a8a3af7f8cb576e533dc6537f14dab105c55

I dug a bit more. It turns out the new certificate has a different intermediate that it's been issued by.

Here's what we have on the current/old cert:

Signer's certificate:
	------------------
	Signer #0:
		Subject: /C=US/ST=California/L=Mountain View/O=Mozilla Corporation/OU=Firefox Engineering Operations/CN=Mozilla Corporation
		Issuer : /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 Assured ID Code Signing CA
		Serial : 0C1CD3EEA47EDDA7A032573B014D0AFD
		Certificate expiration date:
			notBefore : Apr  9 00:00:00 2021 GMT
			notAfter : Jun 19 23:59:59 2024 GMT

And from the new cert:

Signer's certificate:
	------------------
	Signer #0:
		Subject: /C=US/ST=California/L=San Francisco/O=Mozilla Corporation/OU=Firefox Engineering Operations/CN=Mozilla Corporation
		Issuer : /C=US/O=DigiCert, Inc./CN=DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1
		Serial : 0737B0D0DCDCAB8D78D2F40CB122F93F
		Certificate expiration date:
			notBefore : Apr  2 00:00:00 2024 GMT
			notAfter : Jun 18 23:59:59 2027 GMT

Message digest algorithm: SHA256

This is problematic for the stub installer as well as the maintenance service.

To reland, I believe that we need to do the following:

• Update the CertIssuerDownload entries in all of the branding .nsi files to ensure that the stub installer can verify the full installer correctly.
• Update the CERTIFICATE_ISSUER and CERTIFICATE_ISSUER_PREVIOUS entries to ensure that the maintenance service can verify the signature on the updater correctly. It's unclear to me whether we need to do this in advance or not. If it must be done in advance, I believe this requires a watershed.
• Update the maintenance service bootstrap installer. I'm quite unclear if this is strictly necessary, seeing as one of the certs it already has is our self signed one for Try, but it seems unlikely to do any harm.

After a bit more examination, it appears that we did not actually break stub installs here due to the fact that the verification is run against one of the signed plugins, not against the full installer itself.

This means that unless we resign that plugin and land it at the same time, that we can't update the issuer pin in the stub.

Requesting tracking because this needs to land for the 127 / 115.12 releases.

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #10)

After a bit more examination, it appears that we did not actually break stub installs here due to the fact that the verification is run against one of the signed plugins, not against the full installer itself.

This means that unless we resign that plugin and land it at the same time, that we can't update the issuer pin in the stub.

This, as it turns out, is also wrong. download.exe is the full installer that we download.

When I tested last night I didn't see any issues with stub installs though, which is a bit confusing - I would've expected those to fail with a certificate check error.

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #12)

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #10)

After a bit more examination, it appears that we did not actually break stub installs here due to the fact that the verification is run against one of the signed plugins, not against the full installer itself.

This means that unless we resign that plugin and land it at the same time, that we can't update the issuer pin in the stub.

This, as it turns out, is also wrong. download.exe is the full installer that we download.

When I tested last night I didn't see any issues with stub installs though, which is a bit confusing - I would've expected those to fail with a certificate check error.

Ah - this was just a fluke of timing. We didn't publish the builds with the new cert until ~8pm eastern yesterday, and my tests were done about 30min prior to that.

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #9)

To reland, I believe that we need to do the following:

• Update the CertIssuerDownload entries in all of the branding .nsi files to ensure that the stub installer can verify the full installer correctly.
• Update the CERTIFICATE_ISSUER and CERTIFICATE_ISSUER_PREVIOUS entries to ensure that the maintenance service can verify the signature on the updater correctly. It's unclear to me whether we need to do this in advance or not. If it must be done in advance, I believe this requires a watershed.
• Update the maintenance service bootstrap installer. I'm quite unclear if this is strictly necessary, seeing as one of the certs it already has is our self signed one for Try, but it seems unlikely to do any harm.

The other "fun" implication of all of this is that unless I'm mistaken, old stub installers will fail to install the latest nightlies. I can think of two ways to deal with this:

• Go back to DigiCert and see if we can get a certificate from the previous intermediate.
• Keep the existing Bouncer entry for the full installer pointing at an installer that's signed with the current certificate, create a new one for the latest installers, and change the bouncer URL in all the necessary places (stub installer, bedrock, various tests and automation, etc.). This would result in out of date installs for users using an old stub installer as well, so it's got major downsides.

Nick - I'd appreciate your perspective on how much we ought to be doing here to avoid busting installs done from old stub installers.

Bug 1079858 is the last time the intermediate changed and we had to deal with something similar.

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #14)

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #9)

To reland, I believe that we need to do the following:

• Update the CertIssuerDownload entries in all of the branding .nsi files to ensure that the stub installer can verify the full installer correctly.
• Update the CERTIFICATE_ISSUER and CERTIFICATE_ISSUER_PREVIOUS entries to ensure that the maintenance service can verify the signature on the updater correctly. It's unclear to me whether we need to do this in advance or not. If it must be done in advance, I believe this requires a watershed.
• Update the maintenance service bootstrap installer. I'm quite unclear if this is strictly necessary, seeing as one of the certs it already has is our self signed one for Try, but it seems unlikely to do any harm.

The other "fun" implication of all of this is that unless I'm mistaken, old stub installers will fail to install the latest nightlies. I can think of two ways to deal with this:

• Go back to DigiCert and see if we can get a certificate from the previous intermediate.
• Keep the existing Bouncer entry for the full installer pointing at an installer that's signed with the current certificate, create a new one for the latest installers, and change the bouncer URL in all the necessary places (stub installer, bedrock, various tests and automation, etc.). This would result in out of date installs for users using an old stub installer as well, so it's got major downsides.

Nick - I'd appreciate your perspective on how much we ought to be doing here to avoid busting installs done from old stub installers.

Oh dear. So, we believe that there are significant numbers of old stub installers floating around, especially from parts of the unattributed funnel; but we can't (easily) identify them because adding to the stub installer telemetry is hard. The ticket to add the stub's version, which is what we need to understand this impact, never got done: https://bugzilla.mozilla.org/show_bug.cgi?id=1811167.

It's also our belief that installing an old version of Firefox (generally with a full installer, not a stub installer) contributes to poor Firefox user retention. Again, we don't know how frequently that experience occurs (but we should have better estimates).

I expect that it's well worth the effort to ask for a cert signed with the same intermediate to avoid this pain, but NI to Romain to see if he can say more here.

If we do have a breaking change, then at least we'll be able to test some theories around old installers :/

In the meantime, could we get a screenshot walkthrough of what the user experience with a certificate that doesn't validate looks like? If it redirects to mozilla.org in some way that's not a terrible outcome for us.

(In reply to Nick Alexander :nalexander [he/him] from comment #16)

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #14)

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #9)

To reland, I believe that we need to do the following:

• Update the CertIssuerDownload entries in all of the branding .nsi files to ensure that the stub installer can verify the full installer correctly.
• Update the CERTIFICATE_ISSUER and CERTIFICATE_ISSUER_PREVIOUS entries to ensure that the maintenance service can verify the signature on the updater correctly. It's unclear to me whether we need to do this in advance or not. If it must be done in advance, I believe this requires a watershed.
• Update the maintenance service bootstrap installer. I'm quite unclear if this is strictly necessary, seeing as one of the certs it already has is our self signed one for Try, but it seems unlikely to do any harm.

The other "fun" implication of all of this is that unless I'm mistaken, old stub installers will fail to install the latest nightlies. I can think of two ways to deal with this:

• Go back to DigiCert and see if we can get a certificate from the previous intermediate.
• Keep the existing Bouncer entry for the full installer pointing at an installer that's signed with the current certificate, create a new one for the latest installers, and change the bouncer URL in all the necessary places (stub installer, bedrock, various tests and automation, etc.). This would result in out of date installs for users using an old stub installer as well, so it's got major downsides.

Nick - I'd appreciate your perspective on how much we ought to be doing here to avoid busting installs done from old stub installers.

Oh dear. So, we believe that there are significant numbers of old stub installers floating around, especially from parts of the unattributed funnel; but we can't (easily) identify them because adding to the stub installer telemetry is hard. The ticket to add the stub's version, which is what we need to understand this impact, never got done: https://bugzilla.mozilla.org/show_bug.cgi?id=1811167.

It's also our belief that installing an old version of Firefox (generally with a full installer, not a stub installer) contributes to poor Firefox user retention. Again, we don't know how frequently that experience occurs (but we should have better estimates).

I expect that it's well worth the effort to ask for a cert signed with the same intermediate to avoid this pain, but NI to Romain to see if he can say more here.

If we do have a breaking change, then at least we'll be able to test some theories around old installers :/

In the meantime, could we get a screenshot walkthrough of what the user experience with a certificate that doesn't validate looks like? If it redirects to mozilla.org in some way that's not a terrible outcome for us.

Thank you for the perspective! I wasn't aware of the poor retention impacts of installing an older version, that's unfortunate :(.

Things moved very quickly here, I believe that Romain and jcristau synced up earlier today. https://docs.google.com/document/d/1Xzlp4wUNjxZPkOrvUq-yRYhDzhh8sMUCxokklqARJNI/edit#heading=h.j8rkaps9bkzi captures all the latest here.

The tl;dr version for this bug is that:

• We absolutely cannot get a certificate that is compatible with the current pins (we explored every angle of this with DigiCert)
• We have ideas about how we can avoid breaking old stub installers entirely (see the doc)

https://hg.mozilla.org/mozilla-central/rev/c35b87066434

Backed out from central for causing bug 1896944: https://hg.mozilla.org/mozilla-central/rev/b7871b7f2c05e4076ad55ae4aa929b9862d4c8af

https://hg.mozilla.org/mozilla-central/rev/8c5d426162aa

We'll land this for 115.13.0esr, unless there's a dot release for 115.12; updating tracking flag.

This switches us away from the code signing certificate expiring in
June, to a new one valid until 2027.

Update subject pin in msix repackaging and issuer pin in the stub installer.

Original Revision: https://phabricator.services.mozilla.com/D208633

esr115 Uplift Approval Request

• User impact if declined: we can't ship firefox
• Code covered by automated testing: no
• Fix verified in Nightly: yes
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: part of QA-2411
• Risk associated with taking this patch: low
• Explanation of risk level: verified in nightly and beta.
• String changes made/needed: n/a
• Is Android affected?: no

This switches us away from the code signing certificate expiring in
June, to a new one valid until 2027.

Update subject pin in msix repackaging and issuer pin in the stub installer.

Original Revision: https://phabricator.services.mozilla.com/D208633

release Uplift Approval Request

• User impact if declined: we can't ship firefox after this wednesday
• Code covered by automated testing: no
• Fix verified in Nightly: yes
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: part of QA-2411
• Risk associated with taking this patch: medium
• Explanation of risk level: We've had a number of surprises due to bugs in the updater and/or maintenance service and windows msix support, but they're either resolved or deemed acceptable, and we don't really have a choice
• String changes made/needed: n/a
• Is Android affected?: no

Verified that on Windows machines updates on release-cdntest channel from earlier versions of Firefox will go through the watershed 127.0 and then to the Dot Release 127.0.1 - build ID 20240618110440. Verified that the new build 127.0.1 is signed with the new certificate:

Testing was done on Windows 10 and Windows 11, using different installer builds (.msi, .zip, .exe). More details here - https://docs.google.com/spreadsheets/d/1cn38geXHsRDd1N0Gzp3itFTRaeNcyiQ3KkCAOxFZLTc/edit#gid=1263313610

We'll continue verification also on release channel.

Verified again all scenarios on release channel on Windows machines: from earlier versions of Firefox, updates will go through the watershed 127.0 and then to the Dot Release 127.0.1 - build ID 20240618110440. Verified that the new build 127.0.1 is signed with the new certificate: DigiCert Trusted G4 Code Signing RSA4096 SHA384 2021 CA1

Please note that during verification we reproduced bug 1901954, but managed to "skip" it with a machine restart.

Testing was done on Windows 10 and Windows 11, using different installer builds (.msi, .zip, .exe). More details here - https://docs.google.com/spreadsheets/d/1cn38geXHsRDd1N0Gzp3itFTRaeNcyiQ3KkCAOxFZLTc/edit#gid=1353011614

Verified that on Windows machines updates on esr-localtest channel from earlier versions of Firefox will go through the watershed 115.12.0 and then to the 115.13.0 - build ID 20240520161453. Verified that the new build 115.13 is signed with the new certificate:
Testing was done on Windows 7, Windows 10 and Windows 11, using different installer builds (.msi, .zip, .exe). More details here - https://docs.google.com/spreadsheets/d/1cn38geXHsRDd1N0Gzp3itFTRaeNcyiQ3KkCAOxFZLTc/edit#gid=828269799

Also testing .msix build from archive.mozilla.org
for Windows 10 could not install 115.13.0esr over 115.12.0esr: bug 1897155
for Windows 11 could when we try to paveover a 115.13.0esr over an older version, a new instance of firefox is installed: bug 1897155

Verified that on Windows machines the new build 115.13 is signed with the new certificate and updates on esr-cdntest and esr channel from earlier versions of Firefox will go through the watershed 115.12.0 and then to the 115.13.0 - build 20240703222632. More details here -> https://docs.google.com/spreadsheets/d/1cn38geXHsRDd1N0Gzp3itFTRaeNcyiQ3KkCAOxFZLTc/edit?gid=1431089023#gid=1431089023